feat: separate Dockerfile for Hadoop #1186
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maltesander
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
maltesander
reviewed
Jul 1, 2025
…into feat/separate-hadoop-dockerfile
…stackabletech/docker-images into feat/separate-hadoop-dockerfile
dervoeti
moved this from Development: In Review
to Development: Done
in Stackable Engineering
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Follow up for #1173
This PR looks like a lot but it basically just does one thing:
We had a single Dockerfile for Hadoop. This PR separates the build from Hadoop itself from the image build (which also includes things like hdfs-utils). We do the same thing for HBase already, were we have one Dockerfile for the image and another one just to build the HBase JARs.
For Hadoop, this has two advantages:
/stackable/patched-libs, which contains the Hadoop libraries other products use as dependencies. These are not required in the HDFS image though. But because other projects depend onhadoopand need toCOPYthese libraries form the image in order to use them as dependencies, they had to be part of the HDFS image. Now products can just depend onhadoop/hadoopinstead, so just on the Java build itself.In the future we could even think about not copying everything from
hadoop/hadoopintohadoop, but just the components required to run HDFS.One other change:
Since #1173 we build everything of Hadoop, before this PR we skipped Yarn, Mapreduce and Minicluster. These components should be part of the libraries, so that other products can use them, but they should not be part of the HDFS image (they weren't before, now they are). So we just remove them from the distributed JARs after the build now.
I tested building Hadoop 3.3.6 and 3.4.1, as well as Spark 3.5.6, Hive 4.0.0 and Druid 33.0.0. I also ran the smoke tests for the built images. Everything succeeded.
Definition of Done Checklist
Note
Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.
Please make sure all these things are done and tick the boxes
TIP: Running integration tests with a new product image
The image can be built and uploaded to the kind cluster with the following commands:
See the output of
baketo retrieve the image tag for<image-tagged-with-the-major-version>.